{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Performance timings data generation\n",
    "\n",
    "We need to generate data comparing performance of the reference implementation of HDBSCAN and various historical versions of the hdbscan library. We need to do this varying over dataset size so we can get an idea of scaling, and we also need to consider various dimension sizes. To get all this done we'll need some handy modules: sklearn.datasets to generate fake data for clustering; numpy and pandas for easy manipulation of vectors and dataframes of results; and subprocess and time so we can actually fork off and time the actual Java refeence implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import sklearn.datasets\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import subprocess\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we need a function to actually time the reference implementation. We can do external timing use the time module, and the Java program also returns internal timings, which we can parse out and save to a dataframe. In practice this is just a matter of using subprocess an the appropriate commandline parameters for the reference code. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def get_reference_timings(data, filename='tmp_data.csv', \n",
    "                          jarfile='/Users/leland/Source/HDBSCAN_Star/HDBSCAN_Star.jar',\n",
    "                          min_points=5, min_cluster_size=5):\n",
    "    \n",
    "    # Create the required csv file\n",
    "    pd.DataFrame(data).to_csv('tmp_data.csv', header=False, index=False)\n",
    "    \n",
    "    # Run the clustering via a subprocess call and grab the output as it\n",
    "    # has timing information to be parsed\n",
    "    start_time = time.time()\n",
    "    internal_timing = subprocess.check_output(['java', '-jar', jarfile,\n",
    "                                               'file={}'.format(filename),\n",
    "                                               'minPts={}'.format(min_points),\n",
    "                                               'minClSize={}'.format(min_cluster_size),\n",
    "                                               'compact=true'])\n",
    "    time_taken = time.time() - start_time\n",
    "    \n",
    "    # Parse internal timing info into a pandas series for later use\n",
    "    result_dict = {}\n",
    "    for line in internal_timing.split('\\n'):\n",
    "        if ':' in line:\n",
    "            key, value = line.split(':')\n",
    "            key = key.replace(' (ms)', '')\n",
    "            key = key.replace('Time to ', '')\n",
    "            key = key.replace('Overall ', '')\n",
    "            value = int(value)\n",
    "            result_dict[key] = value\n",
    "            \n",
    "    internal_timing = pd.Series(result_dict)\n",
    "    \n",
    "    return time_taken, internal_timing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With that in hand we can run the code over a range of dimensions and dataset sizes and aggregate the results together in indexed pandas series or dataframes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "internal_timing = {}\n",
    "external_timing = {}\n",
    "\n",
    "for dataset_dimension in (2,5,10,25,50):\n",
    "    for dataset_size in np.arange(1,17) * 8000:\n",
    "        data, _ = sklearn.datasets.make_blobs(dataset_size, \n",
    "                                              n_features=dataset_dimension, \n",
    "                                              centers=dataset_dimension)\n",
    "        (external_timing[(dataset_dimension, dataset_size)], \n",
    "         internal_timing[(dataset_dimension, dataset_size)]) = get_reference_timings(data)\n",
    "    \n",
    "internal_timing_df = pd.DataFrame(internal_timing).T\n",
    "external_timing_series = pd.Series(external_timing)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it is just a matter of saving these off to disk for later use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "internal_timing_df.to_csv('reference_impl_internal_timings.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "external_timing_series.to_csv('reference_impl_external_timings.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we need to build up hdbscan timings, preferably over a range of hdbscan versions to show how the performance of the code has evolved (and improved!). To do this I pulled down historical versions and fudged them so that they exist in different namespaces and can live side by side. We can import them all like so ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import hdbscan01\n",
    "import hdbscan02\n",
    "import hdbscan03\n",
    "import hdbscan04\n",
    "import hdbscan05\n",
    "import hdbscan"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we simply go through each version and run over a range of dimensions and dataset sizes (ranging up to smaller sizes in the case of early versions which were memory constrained)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "hdbscan01_timings = {}\n",
    "\n",
    "for dataset_dimension in (2,5,10,25,50):\n",
    "    for dataset_size in np.arange(1,17) * 2000:\n",
    "        data, _ = sklearn.datasets.make_blobs(dataset_size, \n",
    "                                              n_features=dataset_dimension, \n",
    "                                              centers=dataset_dimension)\n",
    "        start_time = time.time()\n",
    "        hdbscan01.HDBSCAN().fit(data)\n",
    "        time_taken = time.time() - start_time\n",
    "        hdbscan01_timings[(dataset_dimension, dataset_size)] = time_taken\n",
    "        \n",
    "hdbscan01_timings_series = pd.Series(hdbscan01_timings).T\n",
    "hdbscan01_timings_series.to_csv('hdbscan01_timings.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "hdbscan02_timings = {}\n",
    "\n",
    "for dataset_dimension in (2,5,10,25,50):\n",
    "    for dataset_size in np.arange(1,17) * 2000:\n",
    "        data, _ = sklearn.datasets.make_blobs(dataset_size, \n",
    "                                              n_features=dataset_dimension, \n",
    "                                              centers=dataset_dimension)\n",
    "        start_time = time.time()\n",
    "        hdbscan02.HDBSCAN().fit(data)\n",
    "        time_taken = time.time() - start_time\n",
    "        hdbscan02_timings[(dataset_dimension, dataset_size)] = time_taken\n",
    "        \n",
    "hdbscan02_timings_series = pd.Series(hdbscan02_timings).T\n",
    "hdbscan02_timings_series.to_csv('hdbscan02_timings.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "hdbscan03_timings = {}\n",
    "\n",
    "for dataset_dimension in (2,5,10,25,50):\n",
    "    for dataset_size in np.arange(1,17) * 4000:\n",
    "        data, _ = sklearn.datasets.make_blobs(dataset_size, \n",
    "                                              n_features=dataset_dimension, \n",
    "                                              centers=dataset_dimension)\n",
    "        start_time = time.time()\n",
    "        hdbscan03.HDBSCAN().fit(data)\n",
    "        time_taken = time.time() - start_time\n",
    "        hdbscan03_timings[(dataset_dimension, dataset_size)] = time_taken\n",
    "        \n",
    "hdbscan03_timings_series = pd.Series(hdbscan03_timings).T\n",
    "hdbscan03_timings_series.to_csv('hdbscan03_timings.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "hdbscan04_timings = {}\n",
    "\n",
    "for dataset_dimension in (2,5,10,25,50):\n",
    "    for dataset_size in np.arange(1,17) * 8000:\n",
    "        data, _ = sklearn.datasets.make_blobs(dataset_size, \n",
    "                                              n_features=dataset_dimension, \n",
    "                                              centers=dataset_dimension)\n",
    "        start_time = time.time()\n",
    "        hdbscan04.HDBSCAN().fit(data)\n",
    "        time_taken = time.time() - start_time\n",
    "        hdbscan04_timings[(dataset_dimension, dataset_size)] = time_taken\n",
    "        \n",
    "hdbscan04_timings_series = pd.Series(hdbscan04_timings).T\n",
    "hdbscan04_timings_series.to_csv('hdbscan04_timings.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "hdbscan05_timings = {}\n",
    "\n",
    "for dataset_dimension in (2,5,10,25,50):\n",
    "    for dataset_size in np.arange(1,17) * 8000:\n",
    "        data, _ = sklearn.datasets.make_blobs(dataset_size, \n",
    "                                              n_features=dataset_dimension, \n",
    "                                              centers=dataset_dimension)\n",
    "        start_time = time.time()\n",
    "        hdbscan05.HDBSCAN().fit(data)\n",
    "        time_taken = time.time() - start_time\n",
    "        hdbscan05_timings[(dataset_dimension, dataset_size)] = time_taken\n",
    "        \n",
    "hdbscan05_timings_series = pd.Series(hdbscan05_timings).T\n",
    "hdbscan05_timings_series.to_csv('hdbscan05_timings.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally we can run the current code (soon to be version 0.6)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "hdbscan06_timings = {}\n",
    "\n",
    "for dataset_dimension in (2,5,10,25,50):\n",
    "    for dataset_size in np.arange(1,17) * 8000:\n",
    "        data, _ = sklearn.datasets.make_blobs(dataset_size, \n",
    "                                              n_features=dataset_dimension, \n",
    "                                              centers=dataset_dimension)\n",
    "        start_time = time.time()\n",
    "        hdbscan.HDBSCAN().fit(data)\n",
    "        time_taken = time.time() - start_time\n",
    "        hdbscan06_timings[(dataset_dimension, dataset_size)] = time_taken\n",
    "        \n",
    "hdbscan06_timings_series = pd.Series(hdbscan06_timings).T\n",
    "hdbscan06_timings_series.to_csv('hdbscan06_timings.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we will save the analysis of all of this for another (rather more text heavy) notebook."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
